Incorporating Invariances in Support Vector Learning Machines

Developed only recently, support vector learning machines achieve high generalization ability by minimizing a bound on the expected test error; however, so far there existed no way of adding knowledge about invariances of a classiication problem at hand. We present a method of incorporating prior knowledge about transformation invari-ances by applying transformations to support vectors, the training examples most critical for determining the classiication boundary. In many applications of learning procedures, prior knowledge about properties of the function to be learned is available (for a review, see Abu{Mostafa, 1995). For instance, certain transformations of the input could be known to leave function values unchanged. Mostly, two diierent ways of exploiting this knowledge have been used: either the knowledge is directly incorporated in the algorithm, or it is used to generate artiicial training examples (\virtual exam-ples") by transforming the training examples accordingly. In the rst case, an additional term in an error function can force a learning machine to construct a function with the desired invariances (Simard et al., 1992); alternatively the invariance can be achieved by using an appropriate distance measure in the pattern space (Simard, Le Cun, and Denker, 1993). The latter is akin to changing the representation of the data by rst mapping them into a more suitable space; an approach pursued for instance by Segman, Rubinstein, & Zeevi (1992), or Vetter & Poggio (1996). In the second case, it is hoped that given suucient time, the learning machine will extract the invariances from the artiicially enlarged training data. Figure 1 contains illustrations of the diierent approaches. Simard et al. (1992) compare the two techniques and nd that for the considered problem | learning a function with three plateaus where the function values are locally invariant | training on the artiicially enlarged data set is sig-niicantly slower, due to both correlations in the artiicial data and the increase in training set size. Moving to real{world applications, the latter factor becomes even more important. If the size of a training set is multiplied by a number of desired invariances (by generating a corresponding number of artiicial examples for each training pattern), the resulting training set can get rather large